
            <!DOCTYPE html>
            <html lang="en">
            <head>
                <meta charset="UTF-8">
                <title>基于K8S构建企业级Jenkins CICD平台实战（一） 之 环境搭建</title>
            </head>
            <body>
            <a href="https://andyoung.blog.csdn.net">原作者博客</a>
            <div id="content_views" class="markdown_views prism-atom-one-light">
                    <svg xmlns="http://www.w3.org/2000/svg" style="display: none;">
                        <path stroke-linecap="round" d="M5,0 0,2.5 5,5z" id="raphael-marker-block" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0);"></path>
                    </svg>
                    <h2><a id="__0"></a>一. 持续集成/部署/交付概述</h2> 
<p><strong>持续集成</strong>(Continuous Integration，CI): 代码合并、构建、部署、测试都在一起，不断地执行这个过程，并对结果反馈。<br> <strong>持续部署</strong>(Continuous Deployment，CD): 部署到测试环境、预生产环境、生产环境。<br> <strong>持续交付</strong>(Continuous Delivery，CD): 将最终产品发布到生产环境，给用户使用。</p> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/309a26730247869cd1e1815db86b3d5e.png" alt="img"></p> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/eb8a47c92a950bc9b33b348fe0bdc023.png" alt="img"></p> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/c3731269da8e32df0a005510f7da047b.png" alt="img"></p> 
<h5><a id="_22"></a>环境配置</h5> 
<table><thead><tr><th>角色</th><th>IP</th><th>备注</th><th>推荐配置</th></tr></thead><tbody><tr><td>K8S</td><td>10.40.6.201 10.40.6.210 10.40.6.213</td><td>自行准备</td><td>CPU:2C+ 内存:4G+</td></tr><tr><td>Harbor</td><td>10.40.6.165</td><td></td><td></td></tr><tr><td>Git</td><td>10.40.6.165</td><td></td><td></td></tr><tr><td>Jenkins</td><td>部署在K8S平台</td><td></td><td></td></tr></tbody></table> 
<h2><a id="__31"></a>二. 准备工作</h2> 
<h3><a id="1__33"></a>1. 对项目的理解</h3> 
<p>• 单体架构、微服务<br> • 怎么部署<br> • 启动是否有依赖</p> 
<h3><a id="2_k8s_39"></a>2. 部署到k8s平台流程</h3> 
<p>(1). 制作镜像<br> (2). 容器放到Pod<br> (3). 控制器管理Pod<br> (4). 暴露应用<br> (5). 对外发布应用<br> (6). 日志管理/监控</p> 
<h3><a id="3__48"></a>3. 不同环境区分配置文件</h3> 
<p>• configmap<br> • entrypoint.sh<br> • 统一配置中心，例如 Apollo，Nacos</p> 
<h3><a id="4_Harbor_54"></a>4. Harbor镜像仓库</h3> 
<p>Harbor 部署参考: <a href="https://blog.csdn.net/agonie201218/article/details/115691894">https://blog.csdn.net/agonie201218/article/details/115691894</a></p> 
<p>kubernetes从harbor拉取镜像没有权限解决方法 unauthorized <a href="https://andyoung.blog.csdn.net/article/details/120313644" rel="nofollow"> https://andyoung.blog.csdn.net/article/details/120313644</a></p> 
<h3><a id="5_GitLab_60"></a>5. GitLab代码版本仓库</h3> 
<p>GitLab 详细安装及使用教程: <a href="https://blog.csdn.net/agonie201218/article/details/120859712">https://blog.csdn.net/agonie201218/article/details/120859712</a></p> 
<h2><a id="_KubernetesJenkins_66"></a>三. Kubernetes中部署Jenkins</h2> 
<p>参考 github jenkinsci/kubernetes-plugin 插件 :<a href="https://github.com/jenkinsci/kubernetes-plugin/tree/master/src/main/kubernetes">https://github.com/jenkinsci/kubernetes-plugin/tree/master/src/main/kubernetes</a></p> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/ea0cc6482e997942e569ccc7072a016c.png" alt="image-20211209155806474"></p> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/abb7997a78c7fbab8d2f02ba182e8454.png" alt="img"></p> 
<p>NFS服务部署 PV、 PVC 参考：<a href="https://andyoung.blog.csdn.net/article/details/120147604" rel="nofollow">https://andyoung.blog.csdn.net/article/details/120147604</a></p> 
<p><a href="https://andyoung.blog.csdn.net/article/details/120843117" rel="nofollow">https://andyoung.blog.csdn.net/article/details/120843117</a></p> 
<h3><a id="1_jenkins_Pod_84"></a>1. 部署有状态的jenkins Pod</h3> 
<pre><code class="prism language-csharp"><span class="token preprocessor property"># jenkins</span>

<span class="token operator">--</span><span class="token operator">-</span>
apiVersion<span class="token punctuation">:</span> apps<span class="token operator">/</span><span class="token class-name">v1</span>
kind<span class="token punctuation">:</span> <span class="token class-name">StatefulSet</span>
metadata<span class="token punctuation">:</span>
  name<span class="token punctuation">:</span> <span class="token class-name">jenkins</span>
  labels<span class="token punctuation">:</span>
    name<span class="token punctuation">:</span> <span class="token class-name">jenkins</span>
spec<span class="token punctuation">:</span>
  selector<span class="token punctuation">:</span>
    matchLabels<span class="token punctuation">:</span>
      name<span class="token punctuation">:</span> <span class="token class-name">jenkins</span>
  serviceName<span class="token punctuation">:</span> <span class="token class-name">jenkins</span>
  replicas<span class="token punctuation">:</span> <span class="token number">1</span>
  updateStrategy<span class="token punctuation">:</span>
    type<span class="token punctuation">:</span> <span class="token class-name">RollingUpdate</span>
  template<span class="token punctuation">:</span>
    metadata<span class="token punctuation">:</span>
      name<span class="token punctuation">:</span> <span class="token class-name">jenkins</span>
      labels<span class="token punctuation">:</span>
        name<span class="token punctuation">:</span> <span class="token class-name">jenkins</span>
    spec<span class="token punctuation">:</span>
      terminationGracePeriodSeconds<span class="token punctuation">:</span> <span class="token number">10</span>
      serviceAccountName<span class="token punctuation">:</span> <span class="token class-name">jenkins</span>
      containers<span class="token punctuation">:</span>
        <span class="token operator">-</span> name<span class="token punctuation">:</span> <span class="token class-name">jenkins</span>
          image<span class="token punctuation">:</span> jenkins<span class="token operator">/</span>jenkins<span class="token punctuation">:</span>lts<span class="token operator">-</span><span class="token class-name">alpine</span>
          imagePullPolicy<span class="token punctuation">:</span> <span class="token class-name">Always</span>
          ports<span class="token punctuation">:</span>
            <span class="token operator">-</span> containerPort<span class="token punctuation">:</span> <span class="token number">8080</span>
            <span class="token operator">-</span> containerPort<span class="token punctuation">:</span> <span class="token number">50000</span>
          resources<span class="token punctuation">:</span>
            limits<span class="token punctuation">:</span>
              cpu<span class="token punctuation">:</span> <span class="token number">1</span>
              memory<span class="token punctuation">:</span> 1Gi
            requests<span class="token punctuation">:</span>
              cpu<span class="token punctuation">:</span> <span class="token number">0.5</span>
              memory<span class="token punctuation">:</span> 500Mi
          env<span class="token punctuation">:</span>
            <span class="token operator">-</span> name<span class="token punctuation">:</span> <span class="token class-name">LIMITS_MEMORY</span>
              valueFrom<span class="token punctuation">:</span>
                resourceFieldRef<span class="token punctuation">:</span>
                  resource<span class="token punctuation">:</span> <span class="token class-name">limits<span class="token punctuation">.</span>memory</span>
                  divisor<span class="token punctuation">:</span> 1Mi
            <span class="token operator">-</span> name<span class="token punctuation">:</span> JAVA_OPTS
              <span class="token preprocessor property"># value: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85</span>
              <span class="token keyword">value</span><span class="token punctuation">:</span> <span class="token operator">-</span>Xmx$<span class="token punctuation">(</span>LIMITS_MEMORY<span class="token punctuation">)</span>m <span class="token operator">-</span>XshowSettings<span class="token punctuation">:</span>vm <span class="token operator">-</span>Dhudson<span class="token punctuation">.</span>slaves<span class="token punctuation">.</span>NodeProvisioner<span class="token punctuation">.</span>initialDelay<span class="token operator">=</span><span class="token number">0</span> <span class="token operator">-</span>Dhudson<span class="token punctuation">.</span>slaves<span class="token punctuation">.</span>NodeProvisioner<span class="token punctuation">.</span>MARGIN<span class="token operator">=</span><span class="token number">50</span> <span class="token operator">-</span>Dhudson<span class="token punctuation">.</span>slaves<span class="token punctuation">.</span>NodeProvisioner<span class="token punctuation">.</span>MARGIN0<span class="token operator">=</span><span class="token number">0.85</span>
          volumeMounts<span class="token punctuation">:</span>
            <span class="token operator">-</span> name<span class="token punctuation">:</span> jenkins<span class="token operator">-</span><span class="token class-name">home</span>
              mountPath<span class="token punctuation">:</span> <span class="token operator">/</span><span class="token keyword">var</span><span class="token operator">/</span><span class="token class-name">jenkins_home</span>
          livenessProbe<span class="token punctuation">:</span>
            httpGet<span class="token punctuation">:</span>
              path<span class="token punctuation">:</span> <span class="token operator">/</span><span class="token class-name">login</span>
              port<span class="token punctuation">:</span> <span class="token number">8080</span>
            initialDelaySeconds<span class="token punctuation">:</span> <span class="token number">60</span>
            timeoutSeconds<span class="token punctuation">:</span> <span class="token number">5</span>
            failureThreshold<span class="token punctuation">:</span> <span class="token number">12</span> # <span class="token operator">~</span><span class="token number">2</span> <span class="token class-name">minutes</span>
          readinessProbe<span class="token punctuation">:</span>
            httpGet<span class="token punctuation">:</span>
              path<span class="token punctuation">:</span> <span class="token operator">/</span><span class="token class-name">login</span>
              port<span class="token punctuation">:</span> <span class="token number">8080</span>
            initialDelaySeconds<span class="token punctuation">:</span> <span class="token number">60</span>
            timeoutSeconds<span class="token punctuation">:</span> <span class="token number">5</span>
            failureThreshold<span class="token punctuation">:</span> <span class="token number">12</span> # <span class="token operator">~</span><span class="token number">2</span> <span class="token class-name">minutes</span>
      securityContext<span class="token punctuation">:</span>
        fsGroup<span class="token punctuation">:</span> <span class="token number">1000</span>
  volumeClaimTemplates<span class="token punctuation">:</span>
  <span class="token operator">-</span> metadata<span class="token punctuation">:</span>
      name<span class="token punctuation">:</span> jenkins<span class="token operator">-</span>home
      <span class="token preprocessor property"># annotations:</span>
      <span class="token preprocessor property">#   volume.beta.kubernetes.io/storage-class: anything</span>
    spec<span class="token punctuation">:</span>
      accessModes<span class="token punctuation">:</span> <span class="token punctuation">[</span> <span class="token string">"ReadWriteOnce"</span> <span class="token punctuation">]</span>
      resources<span class="token punctuation">:</span>
        requests<span class="token punctuation">:</span>
          storage<span class="token punctuation">:</span> 1Gi

<span class="token operator">--</span><span class="token operator">-</span>
<span class="token preprocessor property">## 创建jenkins service</span>
apiVersion<span class="token punctuation">:</span> <span class="token class-name">v1</span>
kind<span class="token punctuation">:</span> <span class="token class-name">Service</span>
metadata<span class="token punctuation">:</span>
  name<span class="token punctuation">:</span> <span class="token class-name">jenkins</span>
spec<span class="token punctuation">:</span>
  <span class="token preprocessor property"># type: LoadBalancer</span>
  selector<span class="token punctuation">:</span>
    name<span class="token punctuation">:</span> jenkins
  <span class="token preprocessor property"># ensure the client ip is propagated to avoid the invalid crumb issue when using LoadBalancer (k8s &gt;=1.7)</span>
  <span class="token preprocessor property">#externalTrafficPolicy: Local</span>
  ports<span class="token punctuation">:</span>
    <span class="token operator">-</span>
      name<span class="token punctuation">:</span> <span class="token class-name">http</span>
      port<span class="token punctuation">:</span> <span class="token number">80</span>
      targetPort<span class="token punctuation">:</span> <span class="token number">8080</span>
      protocol<span class="token punctuation">:</span> TCP
    <span class="token operator">-</span>
      name<span class="token punctuation">:</span> <span class="token class-name">agent</span>
      port<span class="token punctuation">:</span> <span class="token number">50000</span>
      protocol<span class="token punctuation">:</span> TCP

<span class="token operator">--</span><span class="token operator">-</span>
<span class="token preprocessor property">## 创建jenkins ingress</span>
apiVersion<span class="token punctuation">:</span> networking<span class="token punctuation">.</span>k8s<span class="token punctuation">.</span>io<span class="token operator">/</span><span class="token class-name">v1</span>
kind<span class="token punctuation">:</span> <span class="token class-name">Ingress</span>
metadata<span class="token punctuation">:</span>
  name<span class="token punctuation">:</span> <span class="token class-name">jenkins</span>
  annotations<span class="token punctuation">:</span>
    nginx<span class="token punctuation">.</span>ingress<span class="token punctuation">.</span>kubernetes<span class="token punctuation">.</span>io<span class="token operator">/</span>ssl<span class="token operator">-</span>redirect<span class="token punctuation">:</span> <span class="token string">"true"</span>
    kubernetes<span class="token punctuation">.</span>io<span class="token operator">/</span>tls<span class="token operator">-</span>acme<span class="token punctuation">:</span> <span class="token string">"true"</span>
    <span class="token preprocessor property"># </span><span class="token string">"413 Request Entity Too Large"</span> <span class="token class-name">uploading</span> plugins<span class="token punctuation">,</span> increase client_max_body_size
    nginx<span class="token punctuation">.</span>ingress<span class="token punctuation">.</span>kubernetes<span class="token punctuation">.</span>io<span class="token operator">/</span>proxy<span class="token operator">-</span>body<span class="token operator">-</span>size<span class="token punctuation">:</span> <span class="token number">50m</span>
    nginx<span class="token punctuation">.</span>ingress<span class="token punctuation">.</span>kubernetes<span class="token punctuation">.</span>io<span class="token operator">/</span>proxy<span class="token operator">-</span>request<span class="token operator">-</span>buffering<span class="token punctuation">:</span> <span class="token string">"off"</span>
    <span class="token preprocessor property"># For nginx-ingress controller &lt; 0.9.0.beta-18</span>
    ingress<span class="token punctuation">.</span>kubernetes<span class="token punctuation">.</span>io<span class="token operator">/</span>ssl<span class="token operator">-</span>redirect<span class="token punctuation">:</span> <span class="token string">"true"</span>
    <span class="token preprocessor property"># </span><span class="token string">"413 Request Entity Too Large"</span> <span class="token class-name">uploading</span> plugins<span class="token punctuation">,</span> increase client_max_body_size
    ingress<span class="token punctuation">.</span>kubernetes<span class="token punctuation">.</span>io<span class="token operator">/</span>proxy<span class="token operator">-</span>body<span class="token operator">-</span>size<span class="token punctuation">:</span> <span class="token number">50m</span>
    ingress<span class="token punctuation">.</span>kubernetes<span class="token punctuation">.</span>io<span class="token operator">/</span>proxy<span class="token operator">-</span>request<span class="token operator">-</span>buffering<span class="token punctuation">:</span> <span class="token string">"off"</span>
spec<span class="token punctuation">:</span>
  ingressClassName<span class="token punctuation">:</span> <span class="token class-name">nginx</span>
  rules<span class="token punctuation">:</span>
  <span class="token operator">-</span> http<span class="token punctuation">:</span>
      paths<span class="token punctuation">:</span>
      <span class="token operator">-</span> path<span class="token punctuation">:</span> <span class="token operator">/</span>
        pathType<span class="token punctuation">:</span> <span class="token class-name">Prefix</span>
        backend<span class="token punctuation">:</span>
          service<span class="token punctuation">:</span>
            name<span class="token punctuation">:</span> <span class="token class-name">jenkins</span>
            port<span class="token punctuation">:</span>
              number<span class="token punctuation">:</span> <span class="token number">80</span>
    host<span class="token punctuation">:</span> <span class="token class-name">jenkins<span class="token punctuation">.</span>example<span class="token punctuation">.</span>com</span>
  tls<span class="token punctuation">:</span>
  <span class="token operator">-</span> hosts<span class="token punctuation">:</span>
    <span class="token operator">-</span> <span class="token class-name">jenkins<span class="token punctuation">.</span>example<span class="token punctuation">.</span>com</span>
    secretName<span class="token punctuation">:</span> tls<span class="token operator">-</span>jenkins
</code></pre> 
<pre><code class="prism language-undefined">  securityContext:
    fsGroup: 1000  这个配置可以去掉，但得先给后端存储切换组，如果用这个配置，后面jenkins 使用久了，jenkins 重启后会需要很久参能启动成功，这样就会影响健康检测失败，pod一直启动失败
</code></pre> 
<h3><a id="2_jenkins_233"></a>2. jenkins认证授权</h3> 
<pre><code class="prism language-bash"><span class="token comment"># In GKE need to get RBAC permissions first with</span>
<span class="token comment"># kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin [--user=&lt;user-name&gt;|--group=&lt;group-name&gt;]</span>

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: jenkins
rules:
- apiGroups: <span class="token punctuation">[</span><span class="token string">""</span><span class="token punctuation">]</span>
  resources: <span class="token punctuation">[</span><span class="token string">"pods"</span><span class="token punctuation">]</span>
  verbs: <span class="token punctuation">[</span><span class="token string">"create"</span>,<span class="token string">"delete"</span>,<span class="token string">"get"</span>,<span class="token string">"list"</span>,<span class="token string">"patch"</span>,<span class="token string">"update"</span>,<span class="token string">"watch"</span><span class="token punctuation">]</span>
- apiGroups: <span class="token punctuation">[</span><span class="token string">""</span><span class="token punctuation">]</span>
  resources: <span class="token punctuation">[</span><span class="token string">"pods/exec"</span><span class="token punctuation">]</span>
  verbs: <span class="token punctuation">[</span><span class="token string">"create"</span>,<span class="token string">"delete"</span>,<span class="token string">"get"</span>,<span class="token string">"list"</span>,<span class="token string">"patch"</span>,<span class="token string">"update"</span>,<span class="token string">"watch"</span><span class="token punctuation">]</span>
- apiGroups: <span class="token punctuation">[</span><span class="token string">""</span><span class="token punctuation">]</span>
  resources: <span class="token punctuation">[</span><span class="token string">"pods/log"</span><span class="token punctuation">]</span>
  verbs: <span class="token punctuation">[</span><span class="token string">"get"</span>,<span class="token string">"list"</span>,<span class="token string">"watch"</span><span class="token punctuation">]</span>
- apiGroups: <span class="token punctuation">[</span><span class="token string">""</span><span class="token punctuation">]</span>
  resources: <span class="token punctuation">[</span><span class="token string">"events"</span><span class="token punctuation">]</span>
  verbs: <span class="token punctuation">[</span><span class="token string">"watch"</span><span class="token punctuation">]</span>
- apiGroups: <span class="token punctuation">[</span><span class="token string">""</span><span class="token punctuation">]</span>
  resources: <span class="token punctuation">[</span><span class="token string">"secrets"</span><span class="token punctuation">]</span>
  verbs: <span class="token punctuation">[</span><span class="token string">"get"</span><span class="token punctuation">]</span>

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins
subjects:
- kind: ServiceAccount
  name: jenkins
</code></pre> 
<h3><a id="3_jenkins_283"></a>3. 获取jenkins密码并登录</h3> 
<pre><code class="prism language-csharp"><span class="token preprocessor property"># kubectl create -f .</span>
登录容器获取jenkins登录密码：
<span class="token preprocessor property"># kubectl exec -it jenkins-0 bash</span>
bash<span class="token operator">-</span><span class="token number">4.4</span>$ cat <span class="token operator">/</span><span class="token keyword">var</span><span class="token operator">/</span>jenkins_home<span class="token operator">/</span>secrets<span class="token operator">/</span>initialAdminPassword
0f8c730f4b7a426098283cf94aa57231
</code></pre> 
<p>jenkins地址：http://jenkins.example.com:30080</p> 
<h3><a id="4_jenkins__297"></a>4. jenkins 安装插件</h3> 
<ol><li> <p>pipeline</p> </li><li> <p>git</p> </li><li> <p>kubernetes</p> <p><img src="https://i-blog.csdnimg.cn/blog_migrate/54e40967c8d8d076e1e634a3fd4a9089.png" alt="image-20211209161913952"></p> </li><li> <p>Role-based Authorization Strategy(推荐)</p> 
  <blockquote> 
   <p>我们可以利用Role-based Authorization Strategy 插件来管理Jenkins用户权限</p> 
   <p><a href="https://blog.csdn.net/agonie201218/article/details/120168454">https://blog.csdn.net/agonie201218/article/details/120168454</a></p> 
  </blockquote> </li></ol> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/f3be011333d501c113542cddd8f447f9.png" alt="image-20211209161953626"></p> 
<ol start="5"><li> <p>Chinese (中文汉化插件)</p> </li><li> <p>Extended Choice Parameter(参数化构建过程 多选框)</p> <p><img src="https://i-blog.csdnimg.cn/blog_migrate/50af94246cc36fa4eaffe53d3a45d816.png" alt="image-20211209163120882"></p> <p>6.1 多选框：项目名称。设置4个参数，用 , 隔开</p> </li></ol> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/f83d38edae0532c9aa75206a1fc046dc.png" alt="image-20211209164101884"></p> 
<p>6.2 设置多选框值值</p> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/76b668cc21af23aa9601820e9691cd73.png" alt="image-20211209164823212"></p> 
<p>6.3 最终效果</p> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/afb95da2f0bbc0ba6a08a8f311edc0bb.png" alt="image-20211209164916475"></p>
                </div>
            </body>
            </html>
            